[00:00.720 --> 00:08.780]  Oh, okay. Hi, welcome to our next talk. This one's by Gavin Kondyk, also known as GTKondyk on
[00:08.780 --> 00:16.040]  Twitter. He's a pre-circuited researcher, consultant, and has been super involved in
[00:16.040 --> 00:24.060]  the AI village community for a long time. He's an awesome guy, so I'll let him take it away.
[00:27.770 --> 00:34.470]  Hello, and welcome to Hacking with Skynet, How AI is Empowering Adversaries.
[00:34.750 --> 00:39.850]  My name is Gavin. I go by the pseudonym GTKondyk. I am a security researcher
[00:39.850 --> 00:44.410]  and senior security consultant. I'm very passionate about network security,
[00:44.410 --> 00:49.570]  both the attack and defense side. And through that passion, I run a project called NetSecExplained,
[00:49.570 --> 00:55.110]  which is a blog and YouTube channel where I explain intermediate and advanced level concepts
[00:55.110 --> 01:02.870]  in an easy-to-understand way. So, machine learning, let's get everybody on the same page.
[01:03.210 --> 01:10.410]  What is machine learning? Well, first, we have this big umbrella term of AI. And parts of AI
[01:10.410 --> 01:16.350]  are machine learning. Parts of AI are unrelated. So, we have rule-based artificial intelligence.
[01:16.350 --> 01:22.750]  We have classical image recognition. But inside of AI, we have this small subset called machine
[01:22.750 --> 01:28.910]  learning. And machine learning specifically allows us to take statistical analysis and
[01:28.910 --> 01:34.710]  draw inferences from data and perform data discovery techniques. And then out of machine
[01:34.710 --> 01:40.570]  learning, we also have deep learning, which is specifically deep neural networks. And this can
[01:40.570 --> 01:45.750]  be convolutional neural networks, which are used primarily for image recognition, or recurrent
[01:45.750 --> 01:51.210]  neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor.
[01:53.610 --> 02:02.110]  So, the burning question is, how is AI empowering adversaries? Or how can it empower adversaries
[02:02.110 --> 02:09.110]  moving forward? So, I see the age of AI today, or the age of machine learning today,
[02:09.110 --> 02:13.270]  we're at the very beginning of what it can do for the security industry on both the attack and the
[02:13.270 --> 02:20.350]  defense side. Now, if you remember back in the 80s and 90s, antivirus, intrusion detection,
[02:20.350 --> 02:24.570]  vulnerability scanners, these were all simple scripts that were written. And then over time,
[02:24.570 --> 02:30.030]  they grew into what are now multi-billion dollar industries. And so, I think that we're at the very
[02:30.030 --> 02:37.550]  beginning of what AI and machine learning can do for the security community. So, the way that I'm
[02:37.550 --> 02:42.930]  going to bring about this talk is really in two major parts. So, the first is to talk about
[02:42.930 --> 02:49.450]  offensive AI tools. Why do we want these? What's out there already? What are we seeing in the wild?
[02:49.450 --> 02:54.390]  How do we build them? And then adversarial machine learning, which is attacking the machine
[02:54.390 --> 03:01.490]  learning algorithms themselves. So, let's start with offensive AI tools.
[03:04.370 --> 03:10.850]  So, the first question is, why would we even want these? Well, offensive AI tools and AI tools in
[03:10.850 --> 03:17.450]  general will allow us to operate at a much faster speed and a much larger scale than we're currently
[03:17.450 --> 03:22.510]  able to do. And this is going to allow us to automate a lot of manual tasks. Now, we don't
[03:22.510 --> 03:28.370]  need to think about it as automating everything. But if there's something that requires or manages
[03:28.370 --> 03:34.570]  or deals with a lot of data, and these are things that we can apply a certain amount of fuzzy logic
[03:34.570 --> 03:41.210]  to, then this is a perfect candidate for automating with some form of AI or machine learning.
[03:41.370 --> 03:45.690]  And then, of course, this allows us to dynamically and intelligently explore our target attack
[03:45.690 --> 03:52.330]  surface and uncover hidden blind spots that come about some of the biases that we intrinsically have.
[03:52.610 --> 03:57.050]  And so, being able to statistically analyze some of this data or perform some machine learning
[03:57.050 --> 04:00.710]  operations, this will allow us to kind of uncover some of those blind spots
[04:00.710 --> 04:09.710]  and look for things that we don't already know. And then, how realistic is this?
[04:09.710 --> 04:17.930]  Well, back in 2016, DARPA kind of took on a large undertaking in their CyberGram challenge.
[04:18.050 --> 04:23.250]  In here, what they did was they required participants to develop some form of artificial
[04:23.250 --> 04:30.710]  intelligence that will perform a series of attack chain tasks. So, in this case, it was going to be
[04:30.710 --> 04:37.050]  vulnerability discovery, exploitation, patching, and then data theft. So, in the form of data
[04:37.050 --> 04:42.250]  theft, they have capture the flag. And then, as part of their major grand challenge, they had
[04:43.730 --> 04:48.710]  an attack, defend, capture the flag event that was air gapped. And you can see that in the picture
[04:48.710 --> 04:56.230]  kind of off to the side, where there's no human interaction whatsoever for the entire event.
[04:56.230 --> 05:02.950]  And everything had to be automated. So, this was a big reveal on a world stage that let us see
[05:02.950 --> 05:07.790]  what the power of artificial intelligence is, especially when it comes to a number of these
[05:07.790 --> 05:16.170]  tasks. And then, where are we seeing applications of AI already? So, the current proof of concepts
[05:16.170 --> 05:24.930]  we can see for social engineering. We can see in defense detection and evasion. There are some
[05:24.930 --> 05:29.250]  researchers that are identifying intrusion detection systems and sandbox detection systems
[05:29.250 --> 05:36.350]  so that attacks can be stealthier and break out of various systems and bypass existing
[05:36.350 --> 05:42.790]  defense measures. Evaluating data leaks. So, large data sets such as password dumps,
[05:42.790 --> 05:51.010]  subdomains, things of that nature, websites, applications, configuration files. Automated
[05:51.010 --> 05:56.790]  network exploitation. So, this will be going from standard user to domain admin, automating that
[05:56.790 --> 06:04.680]  process, as well as normal internal and external network pen tests. And then, software exploitation.
[06:05.630 --> 06:12.170]  And so, we'll see this with AFL, which is a software fuzzer that will utilize its own custom
[06:12.170 --> 06:20.390]  compiler, put in certain key points in the software itself, and then it allows the system to fuzz and
[06:20.390 --> 06:26.830]  track where through the application user input is being provided and how that can reach potentially
[06:26.830 --> 06:32.790]  vulnerable functions. So, let's start with social engineering. Social engineering, there's a couple
[06:32.790 --> 06:38.970]  things that are in the works right now that will allow us to improve our social engineering
[06:38.970 --> 06:46.290]  capabilities and improve attackers' abilities to exploit corporations and systems. So, in this case,
[06:46.290 --> 06:52.070]  we have an NVIDIA project called StyleGAN. StyleGAN, what it does is generate human faces.
[06:52.710 --> 06:58.970]  So, all four of the images up at the top you see are actually generated by StyleGAN. And then, just
[06:58.970 --> 07:03.610]  to kind of show you that there's no tricks up my sleeve, on the bottom row, we have StyleGAN
[07:03.610 --> 07:10.950]  generated cats. And you can see here in the bottom right corner, this abomination that kind of looks
[07:10.950 --> 07:19.410]  like a cat, but is really messed up and difficult to see, difficult to really recognize for a human.
[07:19.410 --> 07:25.530]  And this is kind of the imperfect model generating a cat-like image. But the level of detail that
[07:25.530 --> 07:28.790]  these can go into, if you look at the cat over here on the left, just kind of sitting on the
[07:28.790 --> 07:34.350]  carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath.
[07:34.950 --> 07:41.810]  And so, these are really difficult to tell whether or not they are legitimate faces or cats, or if
[07:41.810 --> 07:46.450]  these are machine learning generated. One of the telltale signs, if you look here in the top right
[07:46.450 --> 07:51.250]  corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped
[07:51.250 --> 07:57.370]  in the back. You also notice that all of the images have a blurred background. And if you
[07:57.370 --> 08:02.850]  look really carefully, all of the eyes point directly at you. So, it doesn't matter what
[08:02.850 --> 08:09.070]  direction they're faced, the eyes are pointing straight. So, there's already some technologies
[08:09.530 --> 08:13.910]  and some research being performed to kind of identify what is a legitimate human face and
[08:13.910 --> 08:19.410]  what is a generated human face. Now, why is this important? How can we use this for social
[08:19.410 --> 08:25.950]  engineering? Well, for several types of social engineering campaigns, what's pretty common to do
[08:25.950 --> 08:34.250]  is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo,
[08:34.250 --> 08:38.950]  and then you can paste that up as your own LinkedIn profile photo and say, okay, I am now
[08:38.950 --> 08:46.370]  Becky. I'm not a 32-year-old male. My name is Becky. I'm 27. I'm starting my career and I want
[08:46.370 --> 08:51.950]  to ask somebody, hey, as a fellow woman, how did you get started? And then I can use that to build
[08:51.950 --> 08:57.830]  rapport, have my pretext, and perform social engineering tasks to try and exploit employees,
[08:57.830 --> 09:07.090]  maybe in the HR department, sales, wherever. And so, instead of grabbing that image and throwing
[09:07.090 --> 09:12.490]  it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image
[09:12.490 --> 09:18.610]  search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use
[09:18.610 --> 09:24.570]  something like StyleGAN to generate my own face. So, why steal a person when I can just make a
[09:24.570 --> 09:31.950]  whole new person? But there needs to be a little bit more legitimacy to my pretext. So, this is
[09:31.950 --> 09:38.410]  where GPT-2 and Grover come into play. GPT-2 and Grover, they operate very similarly. These are
[09:38.410 --> 09:44.370]  automated generation of long-form content. So, these can be used to generate emails,
[09:44.370 --> 09:51.550]  blog posts, articles, anything of that nature, with very minimal effort from the attacker or
[09:51.550 --> 09:58.110]  the adversary in that respect. And so, unfortunately, the website that was used to generate
[09:58.110 --> 10:04.610]  these has been taken offline, but the GPT-2 model is still something that you can download, put on
[10:04.610 --> 10:11.410]  your own systems, and generate long-form content. So, if I was developing a pretext and I can use
[10:11.410 --> 10:19.690]  this to write phishing emails that could get past a spam filter, or that look distinctly unique,
[10:19.690 --> 10:23.750]  so that I'm not sending the same email to multiple people within an environment,
[10:24.370 --> 10:29.910]  or I can use this to write blog posts and long-form content, and because any sort of
[10:29.910 --> 10:39.010]  Google image search or... what is that called? Plagiarism detection, it will see that I am the
[10:39.010 --> 10:44.210]  only source of truth, therefore I must be a real person, therefore my website must be legitimate.
[10:44.950 --> 10:50.790]  And so, this will kind of add to the pretext. And then on top of that, long-form content Google
[10:50.790 --> 10:56.370]  really enjoys, and so that can give us some Google ranking to add a little bit more credibility to our
[10:56.370 --> 11:04.130]  initial pretext. So, this is one of the scarier ones that I see as being at the very beginning
[11:04.130 --> 11:09.710]  of the research, and I'm curious to see where this will be in the next several years. We have
[11:09.710 --> 11:17.450]  Lyrebird and Tacotron. Lyrebird and Tacotron, these are text-to-speech that are based on human voices
[11:18.130 --> 11:24.970]  and use those human voices to generate conversation bits. And so, we can use this,
[11:24.970 --> 11:30.790]  I foresee, in the phishing campaigns where we're actually calling people or leaving voicemails.
[11:30.790 --> 11:34.950]  So, it's a lot less automated sounding, a little less robocall,
[11:34.950 --> 11:38.730]  and then you just add a little bit of compression and it sounds like you just have a really bad
[11:38.730 --> 11:44.090]  connection. So, I have a couple audio clips. I want you to hear them. So, these first two,
[11:44.090 --> 11:49.670]  I have the original voice and then the second one is the synthetic voice based off that original
[11:49.670 --> 11:57.150]  voice. So, let's go ahead and take a listen. In another moment, down went Alice after it.
[11:57.150 --> 12:01.330]  Never once did she consider how in the world she was to get out again.
[12:03.310 --> 12:09.090]  Awesome. And then the synthetic voice built off of that sample. My voice might be generated by
[12:09.250 --> 12:14.290]  a computer, but I think it sounds pretty human. I don't know exactly how they made it, but I'm
[12:14.290 --> 12:20.110]  really impressed. So, what I'm imagining is that as these models get better, what we can do is
[12:20.110 --> 12:26.590]  take interviews of CEOs and VPs of large corporations, grab their audio, isolate it,
[12:26.590 --> 12:31.930]  and then put it into a model like this so that we can have our own version of synthetic text to
[12:31.930 --> 12:39.890]  speech using their voice. So, we can call throughout the company and impersonate these
[12:39.890 --> 12:46.010]  individuals in order to cause some sort of social engineering action. Now, this last one is really
[12:46.010 --> 12:51.250]  interesting. This is Tacotron. This is a Google project. And I have the text over here on the
[12:51.250 --> 12:55.630]  right, but what I want you to do while you're listening to this is pay attention to the breaths.
[12:55.630 --> 13:01.330]  As humans, we tend to have filler words like um and uh, and then in between sentences and
[13:01.330 --> 13:05.670]  commas, we usually have a bit of a pause or a breath. And so, the Tacotron model actually
[13:05.670 --> 13:11.490]  picked that up. This is fully synthetic voice. Just take a listen. Only the photograph really
[13:11.490 --> 13:16.350]  showed how much time had passed. Ten years ago, there had been lots of pictures of what looked
[13:16.350 --> 13:20.890]  like a large pink beach ball wearing different colored bonnets. But Dudley Dursley was no longer
[13:21.050 --> 13:25.190]  a baby, and now the photograph showed a large blond boy riding his first bicycle
[13:25.750 --> 13:30.390]  on a carousel at the fair playing a computer game with his father being hugged and kissed
[13:30.390 --> 13:40.930]  by his mother by his mother. That's pretty scary. So, these projects are still underway. Liarbird
[13:40.930 --> 13:48.090]  is actually advertising itself as a transcription or reverse transcription service, which is just
[13:48.090 --> 13:53.450]  text-to-speech. In their original pay model that they had pushed out, currently they're in closed
[13:53.450 --> 13:59.610]  beta, but in their original pay model that they pushed out, you can transcribe three hours of
[13:59.610 --> 14:05.650]  audio pretty cheaply. And then if you're so inclined to pay, you can transcribe obviously
[14:05.650 --> 14:11.130]  more. But if a malicious user or somebody who's maliciously inclined, they can just grab those
[14:11.130 --> 14:16.290]  voice samples, throw them into Liarbird or Tacotron, or just use the voice samples that
[14:16.290 --> 14:21.690]  are already in there and generate their own adversarial examples.
[14:24.720 --> 14:30.480]  So, next we get into sandbox detection. And so, this is the defense detection and evasion
[14:31.100 --> 14:36.020]  part of the system. So, we have these researchers over at Silent Brick Security, and what they did
[14:36.020 --> 14:42.640]  was decide that with many different types of sandbox, what you can do is take the process list
[14:42.640 --> 14:48.220]  as data itself. And so, based on the process list, the number of processes,
[14:48.220 --> 14:52.740]  the number of users on a system, you can generate and decide whether or not you're in a sandbox.
[14:52.740 --> 14:57.140]  And then you can take it a step further and decide what kind of sandbox you're in,
[14:57.140 --> 15:06.120]  whether you're in FireEye or Cuckoo sandbox. So, what the researchers did was kind of classify,
[15:06.120 --> 15:11.940]  they took a number of basic Windows systems and they took a number of sandboxes and just kind of
[15:11.940 --> 15:16.600]  identified how many processes there are, how many users there are. Let's take a look at what malware
[15:16.600 --> 15:22.720]  can identify. And so, when we have a low process count, you see over here in column A, we have 33.
[15:22.720 --> 15:29.500]  In column F, we have 34. And then we have a high user count, such as, again, A and F, we have 4
[15:29.500 --> 15:38.320]  and 4. And the process count per user, 8.25, 8.5. It gives us a very high probability that it is a
[15:38.320 --> 15:44.220]  sandbox. And so, we can see that with our host total score. And then based off that total score,
[15:44.220 --> 15:50.780]  we can set a threshold. In this case, it looks like they set the host average score as their
[15:50.780 --> 15:55.280]  threshold. So, anything below the average score is considered a sandbox. Anything above that
[15:55.280 --> 16:01.160]  is considered a legitimate system. And so, they can have malware that completely shuts down or
[16:01.160 --> 16:07.160]  erases itself if it identifies that it's in a sandbox, or it can use targeted evasion techniques
[16:07.160 --> 16:12.880]  per sandbox. So, if I'm in FireEye, these are the things that I'll do to get past it. If I'm in
[16:12.880 --> 16:18.100]  Cuckoo, these are the things that I'll do to get past it. So, it's a really interesting article,
[16:18.100 --> 16:25.260]  highly recommend taking a look at that. And then for automated network exploitation,
[16:25.260 --> 16:30.300]  we have systems like DeepExploit and Goethon. Goethon and DeepExploit, they both operate very
[16:30.300 --> 16:36.020]  similarly. The way that these work is they will go out, they will perform an nmap scan based on
[16:36.020 --> 16:42.100]  the results of that nmap scan, if it sees common services that it knows are exploitable or things
[16:42.100 --> 16:47.400]  that it would like to kind of take to that next level. For example, port 80 or 443, it's web
[16:47.400 --> 16:52.380]  traffic. So, it's going to go ahead and run Scrappy to go and crawl that system. I'm sure they could
[16:52.380 --> 16:58.840]  add other tools such as Go Buster, which will go out and enumerate a lot of the files or directories
[16:58.840 --> 17:03.400]  or URLs that are on that system. Try and fingerprint the server. You want to see if it's running
[17:03.400 --> 17:12.120]  Nginx, Apache, PHP, Java, things of that nature. And then it performs additional content exploration
[17:12.120 --> 17:17.740]  and its own Google hacking, which is kind of cool. So, that will go out, do a little bit of research,
[17:17.740 --> 17:22.920]  see if there's any sort of exploitable scripts, things of that nature. And it takes all of this
[17:22.920 --> 17:31.040]  data and it processes it using some statistical techniques and then it uses Metasploit as a
[17:31.040 --> 17:36.260]  targeted exploitation engine and it goes and tries to exploit those things. So, if it sees
[17:36.260 --> 17:41.120]  certain versions of JBoss or Jenkins, it will go and try and target that with maybe
[17:41.680 --> 17:49.340]  password spray or any sort of Metasploit module. Tomcat, same thing. So, this is kind of cool
[17:49.340 --> 17:55.060]  as a proof of concept of automating the entire process of network discovery all the way up to
[17:55.060 --> 18:02.620]  exploitation and in some cases post-exploitation. One of my favorite while I was researching this
[18:02.620 --> 18:08.940]  is Deathstar. And this is my favorite just kind of in the way that it is assembled. So, Deathstar,
[18:08.940 --> 18:14.160]  it's built on PowerShell Empire. And the way that it works is it starts, it kind of covers its
[18:14.160 --> 18:20.560]  baseline and it is designed to go from standard user all the way up to domain administrator.
[18:20.560 --> 18:26.140]  And so, it charts and maps attack paths all the way to that location. And so, it starts at its
[18:26.140 --> 18:30.780]  baseline and it says, am I already a domain administrator? If so, cancel out. And we see
[18:30.780 --> 18:36.460]  that's the path off to the left. If I have domain administrator credentials, all right, cool. Log
[18:36.460 --> 18:41.420]  in. And that charts off the path to the right. But if it's neither of those, then it goes straight
[18:41.420 --> 18:47.040]  down the middle and it says, okay, let's spawn a new agent. Let's go ahead and identify where is
[18:47.040 --> 18:52.660]  the domain administrators and where is the domain controllers. And so, it runs over here to the
[18:52.660 --> 18:58.160]  right. We can see get net domain controller, get net group member, invoke user hunter. And so,
[18:58.160 --> 19:04.660]  this is going to enumerate the domain controllers, enumerate the domain admins, and enumerate the
[19:04.660 --> 19:09.440]  users that it can on the systems that it can. And you can see it goes through this entire tree
[19:09.440 --> 19:13.660]  kind of off here to the bottom left. It identifies and says, hey, is this a vulnerable,
[19:13.660 --> 19:18.560]  known vulnerable version of Windows, for example, Windows 7? If yes, let's run Mimikatz,
[19:18.560 --> 19:24.040]  try and do Mimikatz, secret stump, things of that nature, collect credentials, and then we can kind
[19:24.040 --> 19:29.420]  of start back over at the top. Let's go ahead and enumerate the running processes. Do we see any
[19:29.420 --> 19:35.440]  processes that are already being ran by other users that we don't have in our inventory? If so,
[19:35.440 --> 19:39.120]  let's add those users to our inventory and try and do some lateral movement.
[19:39.580 --> 19:44.380]  And so, in this case, it'll use psinject, and then it'll go ahead and spawn a new instance.
[19:44.380 --> 19:50.540]  And so, it goes through this iterative process all the way until it reaches domain admin credentials.
[19:51.300 --> 19:57.260]  And so, if you've ever used a system like Bloodhound, where you'll generate an attack
[19:57.260 --> 20:02.640]  graph and say, okay, this user is part of this group, which has these members, which this member
[20:02.640 --> 20:07.360]  is also part of domain admins. And so, this is the attack path that you want to go in order to
[20:07.360 --> 20:13.500]  become domain admin. And so, Deathstar kind of takes that idea a step further and actually tries
[20:13.500 --> 20:22.720]  to navigate that attack path in an automated fashion. So, that's all I have for the machine
[20:22.720 --> 20:27.060]  learning tools that already exist, some of the proof of concepts. There's a lot more information
[20:27.060 --> 20:31.760]  that I wanted to cover that I wouldn't be able to cover in such a short amount of time. So,
[20:31.760 --> 20:37.940]  at the end of this presentation, I have a link to my slide deck, which has a lot more detail
[20:37.940 --> 20:44.280]  and a lot more tools and a lot more techniques in it. But in the meantime, I kind of want to
[20:44.280 --> 20:49.460]  answer the question of how do I go about creating my own machine learning tools
[20:50.360 --> 20:58.180]  or AI tools? What should I look for? How do I automate that? What is that process? And so,
[20:58.180 --> 21:04.100]  honestly, my advice would be to identify somewhere that either utilizes a lot of data
[21:04.660 --> 21:10.900]  or identify somewhere that uses a... it's a common process, but it uses some sort of fuzzy logic,
[21:10.900 --> 21:15.060]  right? Something that's kind of hard to script, hard to write a signature for,
[21:15.060 --> 21:20.400]  but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics
[21:20.400 --> 21:25.660]  and it can perform based off of that. So, in the case of some of these, what we can do is say,
[21:25.660 --> 21:32.060]  all right, NMAP scans, everybody does NMAP scans, and then you have to manually look at them,
[21:32.060 --> 21:37.180]  and then based off of that, you have to follow up and go over here. Instead, like I showed with
[21:37.180 --> 21:43.720]  Goython, is you can just use the NMAP scan and then automate the process of, okay, let's enumerate
[21:43.720 --> 21:47.540]  and fingerprint these services, and then let's see if any of these services are already exploitable
[21:47.540 --> 21:54.240]  using Metasploit, or let's go with next steps. I identified a web service, let's run web analysis
[21:54.240 --> 22:01.820]  techniques. So, Go Buster, Crawler, any sort of things like that. So, that's kind of the thing
[22:01.820 --> 22:06.300]  that you want to keep an eye on if you want to develop your own machine learning tools. And
[22:06.300 --> 22:11.920]  there's more that are coming out all of the time, and it's really interesting and a really fast-paced
[22:12.440 --> 22:17.280]  area, and it's just ripe with research. So, go ahead and play around with some of that.
[22:19.200 --> 22:23.260]  So, the next bit that I want to get into is adversarial machine learning,
[22:23.260 --> 22:28.360]  and this is attacking the machine learning algorithms themselves. So, adversarial machine
[22:28.360 --> 22:33.900]  learning, there's really three ways to go about attacking a machine learning model.
[22:34.160 --> 22:39.780]  So, the first way we have is model evasion. This is to take what is clearly a malicious example
[22:40.300 --> 22:46.500]  and get the model to consider it to be benign or normal or standard to bypass that filter.
[22:46.500 --> 22:50.700]  So, this would be taking a spam message and getting past a spam filter, or this would be
[22:50.700 --> 22:56.780]  taking a piece of malware and getting it past antivirus or anti-malware solutions.
[22:57.420 --> 23:05.160]  Model poisoning is where we add our own data into the training data set. Now, machine learning
[23:05.160 --> 23:13.680]  models specifically are... they're done learning as soon as they're done being trained. To get
[23:13.680 --> 23:18.420]  around this, there are some models that are called online systems, and so what they'll do is they'll
[23:18.980 --> 23:24.860]  take data that has been classified by itself already. So, say I send it a piece of data,
[23:24.860 --> 23:28.680]  like a piece of malware, it looks at that, says, okay, bam, that's malware,
[23:28.680 --> 23:34.820]  now that goes into my training set, and I learn from that malware. So, data poisoning allows us
[23:34.820 --> 23:42.580]  to slowly inject certain pieces of data so that we push that threshold a little off to the side
[23:42.580 --> 23:49.240]  into what is more acceptable, and then we target where that threshold has been pushed.
[23:49.680 --> 23:55.160]  And we'll see an example of that in a bit. Data leakage. Data leakage is actually
[23:56.360 --> 24:03.560]  a combination of two smaller bits. So, one is stealing the model itself. So, for example,
[24:03.560 --> 24:08.820]  say you're an IoT engineering company and you develop an IoT system that utilizes some form
[24:08.820 --> 24:15.000]  of machine learning. What I can do is buy that device and then reverse engineer it,
[24:15.000 --> 24:20.300]  and now I have your model, and so I can use that model or train another model based off of that
[24:20.300 --> 24:23.900]  instead of having to go through the hundreds of thousands of dollars that you went through
[24:23.900 --> 24:30.660]  to develop that in the first place. And so, this is where the model itself is easy to be grabbed.
[24:30.660 --> 24:39.440]  There's also data leakage, which is I can identify what data you use to train the model.
[24:39.760 --> 24:43.240]  And so, this is really damaging, especially for healthcare industry or
[24:45.320 --> 24:50.860]  HIPAA-protected information. So, if I can identify any sort of clients or
[24:52.120 --> 24:57.280]  patients that you have, any sort of their PII, then that's really damaging.
[24:59.760 --> 25:05.080]  So, when it comes to model testing, there's really two approaches. You have a white box where you
[25:05.080 --> 25:08.140]  know what the model is, you know how the hyperparameters are set, you know everything
[25:08.140 --> 25:13.280]  that's involved in the building of this, and then you target that model, much like you would do in
[25:13.280 --> 25:17.820]  software where you already have all of the code involved. And then you have a black box, which is
[25:17.820 --> 25:24.000]  just the model itself. You aren't able to peek into the way that the model operates, but what you
[25:24.000 --> 25:27.260]  can do is treat it like an oracle. So, if you've ever done any sort of cryptography, you know,
[25:27.260 --> 25:34.280]  exercises, you have an oracle, you ask yes-no questions, or things of that nature, and then
[25:34.280 --> 25:40.020]  you can generate what is called a surrogate model. So, based off of the yes-no questions that I asked
[25:40.020 --> 25:45.760]  the black box model, I generated my own model, and then based off of that model, I can perform attacks
[25:46.620 --> 25:52.020]  and that those attacks should be able to impact the original black box itself.
[25:52.660 --> 25:57.780]  Now, the way that this operates is under a principle called attack transferability.
[25:57.900 --> 26:03.160]  This is a documented phenomenon, and the way that attack transferability works is,
[26:03.160 --> 26:08.460]  it doesn't matter what model or the underlying architecture that these models are designed with,
[26:08.460 --> 26:13.600]  an attack on one model can also be an attack on another model. So, if I have a deep learning
[26:13.600 --> 26:18.300]  model over here, and I find adversarial examples that I can use to attack this model,
[26:18.300 --> 26:23.740]  then here is a random forest, a logistic regression, SVM, or anything else, any other
[26:23.740 --> 26:30.220]  machine learning algorithm or architecture, that those same adversarial examples will be able to
[26:30.220 --> 26:37.100]  impact that. And then, of course, the attack surface. So, where are the machine learning
[26:37.100 --> 26:43.980]  models themselves vulnerable? So, machine learning, as it works, is you have a physical object,
[26:43.980 --> 26:48.120]  and then you take that physical object and you try to digitize it. So, say you have a picture
[26:48.120 --> 26:52.840]  of a panda or a car, you take a photograph of it, now it's digitized, and you can make
[26:52.840 --> 26:58.200]  information off of that. You take that digitized information, you run your pre-processing as you
[26:58.200 --> 27:02.540]  would per the normal machine learning process, and then you put that into the machine learning
[27:02.540 --> 27:09.460]  model as part of its input features, it presents some output, usually a probability or a number
[27:09.460 --> 27:16.180]  of some sort, and then a decision is made based off of that action. So, for example, if I am 67%
[27:16.180 --> 27:21.320]  confident that this is malware, then I will probably flag it as malware. If I'm only 42%
[27:21.320 --> 27:27.200]  confident, then I'll just let it continue to operate. So, where can we attack things? Well,
[27:27.200 --> 27:33.620]  one, we can actually impact the physical object itself. So, down below, I have the network
[27:33.620 --> 27:39.020]  construction detection system as kind of an example that follows this generic machine learning model
[27:39.020 --> 27:44.160]  path. So, we have the attack traffic. We can actually, as attackers, modify the attack traffic
[27:44.160 --> 27:49.380]  in such a way that as it goes through the pipeline of the machine learning pipeline,
[27:50.040 --> 27:58.040]  it will modify the behaviors at different points. So, I can slow down my attack traffic,
[27:58.040 --> 28:03.000]  I can use my attack traffic for multiple hosts so that it's a little bit harder to detect.
[28:03.680 --> 28:09.560]  And then that attack traffic is then digitized. The digitization process, we don't really have
[28:09.660 --> 28:17.080]  a lot of impact over as attackers. It's not realistic for us to be able to affect TCP dump.
[28:19.220 --> 28:23.880]  There's really famous examples. For example, you may have come across this if you ever looked at
[28:23.880 --> 28:28.880]  adversarial machine learning, a picture of a panda, and you modify some pixels,
[28:28.880 --> 28:35.260]  and now it thinks it's a gibbon or a bird or some sort. But if we have the ability to modify pixels,
[28:35.260 --> 28:42.120]  why don't we just give it a picture of a bird at that point? And then more to the point,
[28:42.120 --> 28:46.480]  if we can modify pictures, if you have something like a self-driving car, it has a camera,
[28:46.480 --> 28:51.200]  how do we interface with that camera in an adversarial perspective? We would already have
[28:51.200 --> 28:54.860]  to have access to the system, which at that point we can just shut it down or do whatever.
[28:54.860 --> 29:02.820]  It's physical access. So, we're not actually able to interface with the digitization of the object,
[29:02.820 --> 29:08.200]  but what we can do is change the way that the attack traffic is modified.
[29:08.200 --> 29:11.620]  We can change the input features, the input vectors, the machine learning model, especially
[29:11.620 --> 29:16.920]  with the model poisoning. We can change the way that the data is represented.
[29:18.460 --> 29:22.620]  So, in the case of the network intrusion detection system, we have the packet metadata,
[29:22.620 --> 29:28.520]  and then it spits out an attack probability, and then off of that it makes a decision.
[29:28.840 --> 29:32.340]  And in between the machine learning model and the decision-making process, you got to remember
[29:32.340 --> 29:37.540]  this is just software, and you can approach it like any other piece of software that you're
[29:37.540 --> 29:41.440]  pen testing or trying to hack. So, you have the machine learning model in it, and then
[29:41.440 --> 29:45.840]  all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities
[29:46.360 --> 29:54.840]  involved. So, model invasion. Model evasion, not invasion, sorry.
[29:55.680 --> 29:59.600]  So, we're going to try and hide in the blind spots. Now, we have this massive theoretical
[29:59.600 --> 30:05.660]  space. This is all of the possible pieces of malware that could possibly exist, but we can't
[30:05.660 --> 30:11.080]  collect all of that, realistically speaking. Antivirus companies can't collect all of it,
[30:11.080 --> 30:15.520]  and so what they have is a subset, and this is considered the training space. So, this is what
[30:15.520 --> 30:21.140]  they are realistically able to collect. So, in their training space, they perform their training,
[30:21.140 --> 30:25.960]  and then off of a small piece of that that's already clean labeled data, they perform their
[30:25.960 --> 30:32.040]  testing to make sure that the model operates properly. Well, part of that testing space,
[30:32.040 --> 30:35.820]  kind of right there off the sliver, this is the adversarial space. This is where we live.
[30:35.820 --> 30:42.020]  This is where we are able to generate adversarial examples that target that model specifically.
[30:42.440 --> 30:46.760]  And so, this is kind of what we want to play around with and hide in the blind spots,
[30:46.760 --> 30:52.240]  so to speak. So, a couple really good examples is, in this case, multinomial
[30:52.240 --> 30:57.860]  Naive Bayesian spam filters, or just any of the early spam filters that used some rudimentary
[30:57.860 --> 31:02.360]  forms of machine learning, how they would operate is they would look at keywords. So,
[31:02.360 --> 31:09.920]  if you had something like, buy Viagra now, the probability that the word Viagra shows up in a
[31:09.920 --> 31:15.160]  spam email is much higher than the probability of the word Viagra showing up in a regular email.
[31:15.160 --> 31:19.200]  And so, based off of these keywords, it does its own statistical analysis and says,
[31:19.200 --> 31:25.680]  I am 99% sure this is spam. Well, the interesting thing about emails is that
[31:25.680 --> 31:34.440]  emails are usually represented as HTML pages or web pages. And so, as malicious actors,
[31:34.440 --> 31:39.700]  what we can do is actually add in a comment. So, in this case, we can add in the entire Wikipedia
[31:39.700 --> 31:45.200]  page article for a horse. And so, we have a bunch of bad words, but we have even more good words.
[31:45.200 --> 31:51.060]  What this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more
[31:51.060 --> 31:56.960]  good words than bad words, so we must be good, right? It says, yeah, absolutely. I am 99% sure
[31:56.960 --> 32:03.280]  you are not spam. And the thing is that the way that this is presented to users is as an HTML
[32:03.280 --> 32:08.520]  web page. And so, the comment itself is not actually rendered for the users. So, the users
[32:08.520 --> 32:16.620]  still see the original spam email as though it was unmodified. And so, this is kind of a real
[32:16.620 --> 32:23.680]  world example on some of the spam filters. A really good example of this is by Skylight Cyber.
[32:23.680 --> 32:29.200]  The researchers over there were actually able to bypass the silence machine learning antivirus
[32:30.960 --> 32:36.620]  in this article, Silence, I Kill You. Really good article. Highly recommend you take a look at it.
[32:36.620 --> 32:42.060]  We can see some familiar malware samples here. So, we have CoinMiner, Emotet, Zeus,
[32:42.880 --> 32:48.980]  and we see the original scores that they were actually able to pull out of the silence antivirus,
[32:48.980 --> 32:57.920]  negative 826, negative 923, negative 997. So, these are really bad, really bad, flagged immediately.
[32:58.980 --> 33:04.920]  So, what the researchers over at Skylight Cyber did was they took the silence antivirus,
[33:04.920 --> 33:09.920]  reverse engineered it, and they were able to pull out the scores based on some of the
[33:09.920 --> 33:14.740]  syscalls and other key features that silence looked at for malware.
[33:15.940 --> 33:22.120]  One of the curious things is that they identified certain things that were whitelisted. Now,
[33:22.120 --> 33:27.820]  why would silence whitelist certain things, even though the standard machine learning process would
[33:27.820 --> 33:32.220]  actually identify it as potentially malware? Well, the reason why is because when you think
[33:32.220 --> 33:37.720]  about malware, think about how malware operates, right? So, maybe it asks for access to your webcam
[33:37.720 --> 33:45.360]  or your microphone, or it tries to resize or open up new windows. Maybe it makes calls out to the
[33:45.360 --> 33:53.280]  internet. Well, what's another thing that does that? Chrome, Firefox, Internet Explorer. They
[33:53.280 --> 33:59.360]  operate very similar to some modern malware. In this case, what they found was a video game.
[33:59.820 --> 34:04.640]  I'm not sure which video game it is. My money is on Fortnite or League of Legends.
[34:05.240 --> 34:10.520]  But the video games that they found were certain whitelisted keywords. So, what they did was took
[34:10.520 --> 34:17.660]  these clearly bad malware and they just added these keywords to the end of the malware. So,
[34:17.660 --> 34:23.340]  it didn't change the way that the malware operated, but it changed the way that silence looked at it
[34:23.340 --> 34:27.440]  and said, okay, you have a bunch of bad boy points, but you have even more good boy points.
[34:27.440 --> 34:33.020]  So, we're going to give you some good boy scores. So, in this case, CoinMiner had 884.
[34:33.120 --> 34:38.460]  My favorite is looking at Zeus, negative 997, one of the worst in this table,
[34:38.460 --> 34:41.960]  all the way up to positive 997, which is one of the best in this table.
[34:42.680 --> 34:50.140]  And so, it shows that model evasion techniques are very useful in a number of different ways.
[34:52.100 --> 34:55.680]  So, how do we defend against this? Well, the first and probably one of the better
[34:55.680 --> 35:02.220]  is adversarial training. Adversarial training kind of comes from the idea of chaos engineering.
[35:02.220 --> 35:06.600]  Netflix is a really big proponent of chaos engineering, where you already know something
[35:06.600 --> 35:11.500]  bad is going to happen, so you might as well account for those bad things to happen. So,
[35:11.500 --> 35:15.360]  in this case, you train it with adversarial examples. You generate your own adversarial
[35:15.360 --> 35:20.120]  examples and you train the model based off of that to kind of harden and make it a little bit
[35:20.120 --> 35:27.060]  more robust. Again, if it's not a robust model, it's not a good model. Chaos engineering as a
[35:27.060 --> 35:31.820]  whole, the way that it operates is you already know something is going to go bad. So, you have
[35:31.820 --> 35:37.260]  Netflix, for example. Netflix, they know that their systems, they need to have that uptime.
[35:37.260 --> 35:40.280]  And so, they know that it's possible one of their systems goes down. They want to make sure
[35:40.280 --> 35:45.540]  their redundancy operates properly. So, you purposefully go into production and you break
[35:45.540 --> 35:53.360]  things. And then you develop processes and policies around knowing things are going to
[35:53.360 --> 35:58.000]  break. So, in some cases, you'll have the CIO walk into a data center and just start unplugging
[35:58.000 --> 36:05.180]  servers. So, adversarial training is a really good way to kind of counteract this. Another
[36:05.180 --> 36:11.840]  slightly more technical way is called defensive distillation. Defensive distillation is a
[36:11.840 --> 36:16.520]  smoothing algorithm that kind of smooths the curve. So, instead of having, okay, malware,
[36:16.520 --> 36:21.900]  malware, malware, malware, nope, it's okay. You can't have sharp edges like that. So, what it
[36:21.900 --> 36:27.220]  does is it smooths it. So, instead of going straight down, it just kind of goes in a downward
[36:27.220 --> 36:34.680]  trend. And so, it's a little bit harder, a little bit more robust way of analyzing and dealing with
[36:34.680 --> 36:41.400]  the adversarial examples. And then the last example would be monotonic classification.
[36:41.400 --> 36:46.100]  Which means that the graph grows in one direction. So, we see here in the image off to the right,
[36:46.100 --> 36:53.560]  the non-monotonic example, we see that it's getting worse and worse and worse as far as a
[36:53.560 --> 36:57.620]  keyword. So, it goes worse and worse and worse across that threshold. Now, it's considered bad,
[36:57.980 --> 37:02.700]  it's spam, it's malware, it's whatever. We're ready to throw it out, but we're still analyzing
[37:02.700 --> 37:06.680]  it. And so, it's crossed that threshold, but wait, no, it's getting better and better and better.
[37:06.680 --> 37:12.780]  We just hit that pad of keywords that are really positive. And so, we went from good to bad,
[37:12.780 --> 37:18.080]  all the way back up to good. Monotonic classification means that we don't count the
[37:18.080 --> 37:23.280]  good words at all, we say, we measure your badness. How many bad words do you have?
[37:23.560 --> 37:27.680]  And so, in this case, on the bottom example, we have the monotonic classification, which says,
[37:27.680 --> 37:31.420]  okay, we have some bad words, we have some bad words, up, we all went all the way up to the top,
[37:31.420 --> 37:36.620]  and then we hit all of the padded good words. And it doesn't matter, you're still bad,
[37:36.620 --> 37:41.780]  Jesus is not going to save you, you're done. And so, monotonic classification is a really
[37:41.780 --> 37:46.500]  powerful technique to deal with these model evasion techniques.
[37:48.480 --> 37:56.600]  Next, we have model poisoning. Model poisoning almost exclusively works on online learning
[37:56.600 --> 38:01.940]  systems. So, as adversaries, we need to find a way to get adversarial examples into the training
[38:01.940 --> 38:10.240]  space. And so, this is where systems will grab data that it's already classified a certain way,
[38:10.240 --> 38:13.760]  and add it into the training space. And so, we're just trying to push that barrier
[38:15.040 --> 38:20.080]  past a certain threshold where we can actually operate safely as adversaries without it detecting
[38:20.080 --> 38:29.380]  us as such. So, some real world examples. Tay AI, if you're not already familiar,
[38:29.380 --> 38:37.460]  I highly recommend checking it out. Tay AI was a Microsoft Twitter chat bot, and it was designed
[38:37.460 --> 38:46.000]  to learn from real conversations by real people. Well, 4chan, of course, gets a hold of this, and
[38:46.000 --> 38:54.580]  they decide, how much does it learn? How much can we teach it? And about 16 hours after coming online,
[38:54.580 --> 39:01.280]  it was taken offline permanently, because within that time, 4chan was able to get it to say,
[39:01.280 --> 39:08.500]  Hitler did nothing wrong, and racial slurs. If you want more information on Tay AI, and I highly
[39:08.500 --> 39:13.660]  recommend checking this out, it's called The People's Chatbot by the Internet Historian on
[39:13.660 --> 39:20.400]  YouTube. Really funny guy, really great story, and it's just incredible to think of how many
[39:20.400 --> 39:27.240]  data scientists and developers and people behind this project put time in, put effort, put work,
[39:27.240 --> 39:35.880]  and it was shut down by a bunch of kids on 4chan. The other one is Jacobian Map Saliency
[39:35.880 --> 39:43.440]  Attack, and this kind of goes into the pixel manipulation. So, what you do is you grab a
[39:43.440 --> 39:48.540]  couple pixels, and you modify them, or you tweak them a little bit, or you tweak the data a little
[39:48.540 --> 39:54.240]  bit, doesn't necessarily need to be image data, and you pass that on as a certain class.
[39:54.880 --> 39:58.880]  And so, as the model is learning from this data, learning from this class, it starts identifying
[39:58.880 --> 40:05.720]  these tweaks, and then you can implement those tweaks in an adversarial example. So, in this
[40:05.720 --> 40:12.180]  case, we have a 70 kilometer per hour sign, we add in these little tweaks, and it latches onto those
[40:12.180 --> 40:22.120]  tweaks and says, okay, you're a 30 kilometer per hour sign. And so, imagine the kind of chaos that
[40:22.120 --> 40:30.280]  that can create. Again, these tweaks are difficult to implement in real world examples. This is also
[40:30.280 --> 40:36.980]  difficult to pull off simply because it relies on an online learning system, something that learns
[40:36.980 --> 40:41.680]  from the data that you've already presented it. But these are things that researchers have been
[40:41.680 --> 40:45.600]  able to identify and actually use to attack machine learning systems.
[40:47.440 --> 40:53.540]  So, how do we defend against poisoning attacks? Basically, it comes down to, you can still use a
[40:53.540 --> 41:00.040]  lot of the data, but what you want to do is add your own little smoothing filter. So, in this case,
[41:00.040 --> 41:06.620]  have longer periods of time between retraining. So, what that will do is kind of narrow the window,
[41:06.620 --> 41:13.600]  not narrow the window, it will expand the window that an attacker would need to generate adversarial
[41:13.600 --> 41:19.690]  examples for. And then they would also have to compete for votes against legitimate traffic.
[41:20.240 --> 41:27.280]  So, it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of
[41:27.280 --> 41:36.060]  take over cryptocurrency networks. So, this is kind of their own example of doing this.
[41:36.060 --> 41:41.180]  You can also analyze longer periods of data. Again, it broadens the window that you're looking
[41:41.180 --> 41:45.060]  at. And so, the adversaries, the attackers are going to have to generate more traffic,
[41:45.060 --> 41:50.680]  more examples in a longer period of time, which is difficult to do. And then you just want to
[41:50.680 --> 41:54.640]  generally minimize the impact of adversarial training examples. So, whether you have a human
[41:54.640 --> 42:00.960]  in loop that's manually looking at some of these examples, or maybe you're taking a small sample
[42:00.960 --> 42:06.240]  of the stuff that you're seeing and only allowing that to affect the model very...
[42:08.420 --> 42:15.060]  a little bit. So, it's just kind of a general rule of thumb. It's just minimize the impact,
[42:15.060 --> 42:21.900]  come to the understanding that people will try and poison the well, account for that.
[42:23.460 --> 42:29.420]  And then, of course, we have data leakage. So, data leakage, this is usually when,
[42:29.420 --> 42:34.160]  if we're trying to steal data, not steal the model, but steal data itself, this is usually
[42:34.160 --> 42:40.440]  when the models are trained to be a little too good. So, this image down below, you see aerial
[42:40.440 --> 42:45.320]  photographs, generated map, and then an aerial reconstruction. And so, articles at the time,
[42:45.320 --> 42:50.300]  this is a project by Google, articles at the time were kind of grandstanding and saying,
[42:50.300 --> 42:56.900]  oh my god, the Google AI found a way to cheat and generate aerial photographs.
[42:57.460 --> 43:02.640]  It was able to embed its own data. From the machine learning perspective or data science
[43:02.640 --> 43:06.360]  perspective, this is actually called overfitting. It's a very common problem. It's nothing
[43:06.360 --> 43:14.140]  spectacular. It's just, you didn't train your model right. And so, especially with overfit
[43:14.140 --> 43:18.040]  models, you have the training data in the model itself, or the model is able to generate that
[43:18.040 --> 43:23.440]  training data. And so, as I was saying earlier, with things like credit card information, PCI
[43:24.000 --> 43:28.480]  protected information, personally identifiable information, public health, or not public,
[43:28.480 --> 43:34.160]  sorry, personal health information, or HIPAA protected data, things of that nature,
[43:34.160 --> 43:41.680]  it can be very damaging for a machine learning model to leak real training data. And so,
[43:41.680 --> 43:48.500]  it's important to kind of evaluate and make sure that the models are not overfit, are not underfit,
[43:48.500 --> 43:53.700]  just hidden just right, or that the data is being preprocessed in such a way that it's
[43:53.780 --> 43:59.760]  a one directional. So, in some cases, what they'll do is generate a hash of certain features. For
[43:59.760 --> 44:03.800]  example, instead of putting in a credit card number, they generate a hash of the credit
[44:03.800 --> 44:12.140]  card number. And so, it's not reversible in any meaningful way. And then, of course, model theft,
[44:12.140 --> 44:18.340]  model theft by competitors. There's a technique called federated learning, which is you encrypt
[44:18.340 --> 44:26.180]  the machine learning model itself so that the model has to be used in that encrypted form. You
[44:26.180 --> 44:32.620]  can't steal the model, you can't decrypt it. It's a one way kind of thing, or a one way street in
[44:32.620 --> 44:38.680]  that scenario. And so, this will allow like IoT devices to be deployed with machine learning
[44:38.680 --> 44:43.920]  models. And it will prevent competitors or attackers, malicious actors from reversing it,
[44:43.920 --> 44:46.820]  stealing that model and building their own product off of it.
[44:49.420 --> 44:53.760]  And then, one last thing that I kind of want to cover is adversarial stickers. So,
[44:53.760 --> 44:58.720]  you've probably seen adversarial stickers. They look like melted crayons, in my opinion.
[45:00.980 --> 45:05.700]  But adversarial stickers, they have a bit of a problem. This is considered a brittle attack,
[45:05.700 --> 45:11.680]  and it's not very repeatable. And the reason why is because it doesn't have that attack transfer
[45:11.680 --> 45:19.980]  ability that I brought up earlier. So, in this case, kind of off to the bottom right,
[45:19.980 --> 45:24.440]  the stop sign, this is one of the first articles I was talking about adversarial stickers, and they
[45:24.440 --> 45:31.960]  were trying to modify a stop sign in such a way that the machine learning model didn't recognize
[45:31.960 --> 45:36.360]  it as a stop sign anymore, recognized it as something else. And so, this could be really
[45:36.360 --> 45:42.560]  damaging for self-driving vehicles. One of the problems is that this paper ran into some
[45:42.560 --> 45:48.400]  repeatability problems. And the reason why is because this attack is so brittle, they were
[45:48.400 --> 45:54.080]  able to convince their model, but they were not able to convince, or people who were trying to
[45:54.080 --> 46:00.980]  replicate this were not able to convince other models. So, it's not very useful. You can't just
[46:00.980 --> 46:09.440]  slap an adversarial sticker onto a sign and hope that you're driving a self-driving vehicle off
[46:09.440 --> 46:15.480]  into a wall or something of that nature. And then off here to the left is kind of the same idea with
[46:15.480 --> 46:22.360]  facial recognition. So, this researcher here said, okay, I'm going to set some constraints. I only
[46:22.360 --> 46:28.300]  want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like
[46:28.300 --> 46:35.860]  this actress. Honestly, I don't remember who this is. I apologize. But I'm a man. I want to look like
[46:35.860 --> 46:41.560]  this actress per a facial recognition model. And so, he was successfully able to do that.
[46:41.760 --> 46:49.380]  The problem is that these glasses are designed for his face and the rest of the features of his face.
[46:49.420 --> 46:54.440]  And so, if I or many of you put on these glasses, it'll either look like somebody else or it won't
[46:54.440 --> 47:01.740]  recognize it as an adversarial example and it'll just recognize us as people, the same people,
[47:01.740 --> 47:06.620]  just wearing stupid glasses. So, attack transferability, adversarial stickers don't
[47:06.620 --> 47:11.800]  really have it. It's not a really good attack. It's very brittle. And I just kind of wanted to
[47:11.800 --> 47:20.140]  bring that up. For more information on this, I highly recommend DEF CON 2018 AI Village. Sven
[47:20.140 --> 47:25.940]  has a really good 30-minute talk on adversarial stickers and kind of the challenge with them.
[47:25.940 --> 47:33.420]  And then, for the facial recognition specifically, I know that Rich Horang in AI Village 2019 has a
[47:33.420 --> 47:39.580]  really good talk talking about facial recognition and trying to bypass that.
[47:39.580 --> 47:47.180]  So, to recap, how has AI empowered attackers? Well, it allows us to operate at a speed and scale
[47:47.180 --> 47:53.460]  like never before. We can operate at machine speed. And then, what we are seeing is the very
[47:53.460 --> 47:58.540]  beginning of what AI can bring to the offensive security space. So, we've already seen some
[47:58.540 --> 48:04.940]  adversarial attacks in the wild. I anticipate us seeing more as more researchers are coming out.
[48:05.120 --> 48:10.180]  AI and machine learning is kind of the hot new buzzword that everybody's latching onto. So,
[48:10.180 --> 48:15.480]  CEOs and decision makers are grabbing the AI-powered or machine learning-powered
[48:15.480 --> 48:21.000]  detection systems, defense systems. And it's kind of become a bit of a buzzword
[48:22.180 --> 48:27.260]  where you say that in a lot of common companies, especially in places like DEF CON, and they tell
[48:27.260 --> 48:31.980]  you to go talk to the Bitcoin folks and blockchain and all that fun stuff. There's no silver bullet,
[48:31.980 --> 48:37.880]  but we are seeing the very beginning of what is possible. And I anticipate, and I'm really hoping
[48:37.880 --> 48:45.820]  in the next several years, that we see a much larger impact of this AI versus AI kind of warfare
[48:47.480 --> 48:53.720]  that should be coming about. But anyways, I appreciate it. Thank you for your time. Thanks
[48:53.720 --> 48:58.160]  for sticking around this long. You can go ahead and email me if you have any questions. I'm also
[48:58.160 --> 49:04.300]  going to be in the chat. And then, for the slides, I have that are much more detailed are on
[49:04.300 --> 49:10.820]  Slideshare, and you can find them at slideshare.net slash GTKalandai. So that is my presentation. I
[49:10.820 --> 49:16.100]  hope you learned something, and thank you for sticking around. Bye!
